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IN THIS ISSUE: 

Keeping Professional Learning 
on Track With Evaluation 

T he delivery of professional 

development for teachers rarely 
proceeds exactly according to plan. 

Instead, the director of professional 
development must continually adapt 
the overall program to meet the spe- 
cific needs of the school, grade, or 
teacher. In order to make these 
adjustments, the director and 
provider of professional development 
need continuous feedback about the 
effectiveness of the program. For 
example, they need to know if workshops and presentations are addressing 
topics that are important and relevant to teachers and are providing them with 
adequate opportunities to develop new skills. 

At several junctures, it is possible to gather information that is essential to the 
success of professional development. School and district personnel have the 
ability and the need to collect information about the effectiveness of the pro- 
fessional development they plan and conduct. This newsletter provides a prac- 
tical guide for evaluating the effectiveness of all stages of professional devel- 
opment and for using that information to manage the effort. 
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P icture this: A school improvement team at a middle 
school identifies the need to increase student achieve- 
ment in science. After undertaking a careful review of the 
specific areas for improvement, they decide to phase in a 
science curriculum based on an inquiry approach to 
instruction. To make this transition, the school begins 
offering professional development through an external 
service provider. The eight science teachers are enthusias- 
tic about the approach at first, but they soon encounter a 
number of difficulties as they try to implement it. As the 
date for state assessments approaches, the eight teachers 
decide to revert to their former lecture- 
based instructional style in order to “give 
over” the information more efficiently. 

What is going wrong here? 

In this example, both the teachers and the 
professional developer were excited about 
and committed to the stated improvement 
goals but the improvement efforts were 
unsuccessful. Nevertheless, the director of 
professional development could not 
respond to the difficulties of the teachers 
because no one was keeping track of how 
well they were able to implement the train- 
ing. For situations like this, it is essential 
to evaluate professional development to 
ensure that the school is making progress 
towards its goal of improving student 
achievement. 

One reason to evaluate is to learn how to improve the 
effectiveness of the actual professional development ses- 
sion. To begin with, evaluation helps the professional 
development provider adjust the level of difficulty and the 
focus of the workshops or activities to match the needs of 
the participants. Also, evaluation helps verify that teachers 
are learning the intended knowledge and skills. In sum- 
mary, evaluation helps to adjust the learning experience to 



serve the goals of the school and the needs of specific 
teachers. In the words of Dr. Larie Godinez, director of 
professional and program development at Community Unit 
School District 300 in Carpentersville, 
Illinois, “One size doesn’t fit all — not for 
kids and not for adults. [Evaluation] can 
gear the professional development towards 
needs of the adult learners. The more infor- 
mation we have, the better choices we 
make.” 

A second reason for evaluation is to deter- 
mine the effectiveness and impact of profes- 
sional development on teachers and students. 
Impact on teachers can be determined by 
observing the extent and quality of their 
implementation of new techniques in the 
classroom. This observation often uncovers a 
gap in the training or a logistical obstacle 
preventing a teacher from putting the profes- 
sional development into practice. The bot- 
tom line in evaluating impact, however, is whether the pro- 
fessional development had its intended effect on student 
learning. Demonstrating effectiveness encourages partici- 
pating teachers to persist with learning and implementing 
new techniques — and encourages nonparticipating teachers 
to jump on the successful bandwagon. In addition, the par- 
ents and citizens whose taxes support the school will want 
to know whether the cost of professional development is 
money well spent. All of these stakeholders have the 



“One size doesn ’t fit all — 
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gear the professional 
development towards 
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desire — and the right — to be informed about the progress 
of professional development. Although the professional 
development director and facilitators are rarely in the posi- 
tion to document scientific proof of effectiveness, they can 
provide a wealth of information that can offer persuasive 
evidence for its value and worth. 

Unfortunately, many directors and providers of staff devel- 
opment think about collecting pertinent information only 
when they are called on to provide this sort of evidence. 

At this point, however, amassing the evidence that will 
document months or years of professional learning is often 
difficult and costly. Planning ahead for evaluation, in con- 
trast, can make the process much easier and less expensive. 
For this reason, it is helpful to have an overview of the 
entire process of evaluation before beginning any new pro- 
fessional development initiative. 

In his book Evaluating Professional Development, Guskey 
(2000) describes five levels of evaluating professional 
development, which correspond to five expected outcomes 
of professional learning. These outcomes and levels are 
summarized in Table 1 . 

Level 1 is the evaluation of the teachers’ initial reactions to 
the professional development activity. Level 2 is the evalu- 
ation of the immediate impact of the event on teacher 



learning; it focuses on whether teachers have indeed 
learned the intended knowledge and skills of the activity. 
Level 3 is the evaluation of the school organization’s 
efforts to support teachers as they put their learning into 
practice. Level 4 evaluates the implementation of the learn- 
ing, such as the quality and frequency of use in the class- 
room. Finally, Level 5 is the evaluation of whether a suc- 
cessfully implemented program actually improved student 
learning. Although this level is the last to be observed, it 
must be the first in thought: Professional development 
must be directed towards improving student achievement. 
For this reason, professional development should be pre- 
ceded by a careful examination of student data to identify 
those goals, as called for in the Standards for Staff 
Development developed by the National Staff 
Development Council (NSDC, 2001). 

The following sections of this newsletter explain how to 
use evaluation to help manage and improve professional 
development. Many of the techniques described here are 
perfectly suited for the purpose of gathering evidence of 
the impact and value of the professional development as 
well. Because this newsletter is geared towards school-, 
district-, and regional-level leaders, it presents evaluation 
techniques that are relatively simple to conduct. 



Table 1 . Professi 

Professional 

Development 

Outcomes 


onal Development 

Teachers 
participate 
in professional 
development 
activities. 


Outcomes and Levt 

Teachers gain 
new knowledge 
and skills. 


sis of Evaluation 

The organization 
supports the use 
of new knowledge 
and skills. 


Teachers attempt 
to implement 
what they learned. 


Students 
benefit from 
improved 
instruction. 


Levels of 
Evaluation 


Level 1 : 
Evaluate 
teacher 
reactions. 


Level 2: 
Evaluate 
teacher 
learning. 


Level 3: 
Evaluate 
organizational 
support. 


Level 4: 

Evaluate 

classroom 

implementation. 


Level 5: 

Evaluate 

student 

learning 

outcomes. 
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LEVEL 1: EVALUATE TEACHER REACTIONS 



M uch can be learned about the quality and appropri- 
ateness of professional development events and 
activities by evaluating the reactions of participants. The 
point is not to understand the long-term impact but to 
determine if the professional development had the intended 
short-term impact. This basic level of evaluation focuses 
on whether participants liked the event and found it useful 
and worthwhile. This information is far from trivial, 
because teachers are unlikely to participate actively unless 
the professional development meets their needs. 

Topics to Investigate 

Participants can be asked to assess their reactions to the 
content, process, and context (Guskey, 2000) of the profes- 
sional development event, as follows: 

• Content. The content of an event, workshop, and activity 
is the topic it addresses. Did the teachers perceive that the 
topic was relevant to their professional learning goals and 
applicable in their classrooms? Did the presentation assume 
an appropriate level of prior knowledge on the topic? 

• Process. The process of professional development is the 
way in which the material was presented. Did participants 
have the opportunity to participate actively? Did they have 
an opportunity for peer discussion? Was the presenter 
organized and knowledgeable? Was the pace appropriate? 
Were the teachers adequately prepared with background 
reading in order to participate? 



Tip: Identify Participant Background 



At all levels of evaluation, identifying the 
background of the participants themselves — such as 
their subject area, years of experience, and school 
or classroom level — is important. This information 
helps to target the training to specific groups of 
teachers. For example, a principal at an elementary 
school observed that new teachers who had recently 
graduated did not need as many workshops on 
differentiated instruction as other teachers because 
most of them already had received such training 
during their preservice education. 



• Context. The context is background to the event. Were 
the facilities and equipment adequate for supporting the 
learning during this session? 

An important additional aspect to assess is attendance — 
namely, who showed up at the event. Evaluation pros call 
this aspect service utilization. If teachers are not attending 
a service, there is no point in looking at its effectiveness. 
This information needs to be gathered at the event, 
although it can be most useful later on to explain differ- 
ences in the quality of implementation. 

How to Use the Information 

Participant reactions can be used in the following ways to 
make common- sense decisions about the delivery of the 
professional development: 

• Adjust and refine the training. Initial reactions to the 
format of the event and its context can be used to adjust 
or fine-tune the workshop or activity to the needs of the 
participants. It is useful to have participants write their 
reactions on forms, which can be examined in the middle 
of the activity (e.g., after the first day of a two-day event) 
as well as at the end. These reactions can identify areas of 
the presentation where participants need more depth of 
discussion or help with particular skills. For example, par- 
ticipants may request more time on certain topics or 
greater opportunity for peer discussions. 

• Identify unserved groups. Comparing the reactions of 
different groups of participants (e.g., new teachers and 
“veterans”) can identify participants who need a different 
type of service. For example, some teachers may need 
additional workshop time, whereas others may be beyond 
that stage of learning. (See the tip “Identify Participant 
Background” at left.) 

• Provide quality control. Sometimes a district or local 
agency may work with several different providers of the 
same type of service (e.g., three different facilitators of a 
guided reading workshop). Evaluations help ensure that 
sessions conducted by different consultants are consistent 
in quality, according to teacher ratings. 
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• Leant about the school. For the professional development 
provider, reactions help to illuminate the context of the school, 
as reflected by the concerns and needs that participants 
express. 

How to Get the Information 

Surveys. Surveys such as the “Participant Reaction Survey” 
(below), are the typical means to obtain participant reactions. 



Some items on surveys can he fixed response, in which the 
respondents rate statements on a scale (e.g., a 5-point “agree” 
to “disagree” scale). Other items are best to leave as free 
response, in which the respondent answers the question (e.g., 
“What did you like best about the workshop?”) in his or her 
own words. Surveys that assess participant reactions are avail- 
able from a variety of sources. It also is possible to constmct 
specific questions about the workshop. 



Tool: Participant Reaction Survey 

The following participant reaction survey is a brief but useful guide for the format and phrasing of survey questions. 


1. When considering your experience participating in [insert name of professional development], how much 
do you agree or disagree with the statements below? 


Statement 


1 

Strongly 

Disagree 


2 


3 

Neutral 


4 


5 

Strongly 

Agree 


N/A 

or 

Unsure 


A. 1 gained new knowledge and/or skills that 1 can 
apply in my current position, (content) 














B. The content of the sessions addressed real-life job 
situations 1 encounter, (content) 














C. When participating in the sessions, 1 had an 
opportunity to learn from my peers, (process) 














D. The pace of the service was about right, (process) 














E. 1 had adequate help and support from the 
facilitator, (process) 














F. The visual aids (e.g., overheads, videos, 
multimedia presentations) contributed to my 
learning during the service, (context) 














G. The equipment functioned well, posing no 
problem to the learning experience, (context) 














2. Please rate the professional development in terms of the following: 




Excellent 


Good 


Fair 


Marginal 


Poor 


A. Organization of the content 












B. Timeliness of the information for 
my professional practice 












C. Presentation and facilitation 












3. What did you like most about the sessions? 


4. What could be improved about the sessions? 
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Tip: Postevent Surveys 



One of the most useful end-of-session feedback 
questions is to ask whether the session met its 
objectives. The professional development 
facilitator must establish clear objectives for the 
session and to communicate those objectives 
publicly to the participants beforehand. At the end 
of the session, the participants can rate the extent 
to which the session met its objectives, along with 
how much they value the objectives. In this way, 
the facilitator can discover if the participants felt 
that the purpose of the session was important to 
them in the first place. 

Surveys have benefits and drawbacks. On the plus side, 
they are fairly easy to distribute and provide a low-cost 
way to collect information. On the minus side, experts in 
evaluation agree that the collected information is not 
always valid. Some participants may be hesitant to report 
negative reactions to events. Some participants may not be 
motivated to answer the questions completely. For these 
reasons, it is wise not to rely upon survey results alone 
when assessing reactions — or any other stage of profes- 
sional development. 

The following suggestions can be used to increase the rate 
and quality of survey responses: 

• Provide time. Provide time for participants to fill out the 
surveys as part of the workshop. If possible, collect the 
surveys in person. 

• Create accountability for completion. Have participants 
quickly initial a sign-out sheet (i.e., a list of printed names) 
as they hand in their surveys. 

• Ensure confidentiality. Assure teachers that their 
responses are completely confidential and that the only 
purpose for the survey is to help target the professional 
development service to their needs. 

Interviews and Focus Groups. For the school-based pro- 
fessional development director, informal interviews with 
the participants are a useful way to learn about their reac- 
tions. The purpose of these interviews is to get a richer 
sense of how certain participants reacted to the professional 
development services. The format of the questions typically 
is open-ended, such as “How do you feel about the profes- 



sional development?” and “What has been most helpful?” 
Of course, the interviewer should inform the teachers that 
the purpose is merely to adjust the professional develop- 
ment services to the needs of the school. 

Several techniques are helpful for getting participant reac- 
tions with postevent interviews. If the group is not too large 
(e.g., 12 participants or fewer), an informal whole-group 
discussion might be possible. For larger groups, it usually 
is better to create small discussion groups of two to four 
participants and have each group report on their reflections 
orally or in writing. (Additional resources for conducting 
these interviews and focus groups are listed below.) 

Additional Resources for 
Evaluating Teacher Reactions 

Professional Development Modules 

WWW. oerl. sri. com/module/ 

The Online Evaluation Resource Library (OERL) offers 
a series of online professional development modules for 
novice evaluators who want to learn more about 
developing written surveys. 

Teacher Education Instruments 

www.oerl.sri.com/instruments/te/teachwork/ 
instrTE_teachwork. html 

The OERL Web site lists numerous sample instruments 
for evaluating workshops. 

What Are Focus Groups? 

www.ncrel.org/info/notes/fall02/focus.htm 

Notes & Reflections Issue 3 discusses how to conduct 
focus-group interviews. 
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LEVEL 2: EVALUATE TEACHER LEARNING 



T he second level of evaluation is to assess 
whether the participants are learning the 
knowledge and skills that they will need to 
improve their teaching in the desired manner. 

Proper learning of knowledge and skills must 
precede implementation. If learning did not 
take place, the professional development 
activities will need to be refocused or 
expanded. Although this learning may be 
assessed to some degree directly after the 
workshop or event, typically several months 
may be necessary before one can reasonably 
expect participants to have attained many learning goals. 
The careful director of professional development will be 
sure to assess teacher learning continuously from a variety 
of sources (NSDC, 2001). 



Tip: Inform Participants About 
Learning Evaluation 



Informing participants prior to a professional 
development event that they will rate their amount 
of learning at its conclusion often motivates better 
focus and participation. Apparently, this reminder 
sets forth the expectation that teachers will learn 
something, thereby increasing their interest 
(Guskey, 2000). 

Topics to Investigate 

Several types of learning are useful to assess: 

• Teacher Knowledge. Knowledge encompasses specific 
subject content (e.g., deeper conceptual understanding of 
mathematics) as well as the rationale for pedagogical 
approaches (e.g., cooperative learning or use of classroom 
assessment). For teachers, understanding the rationale for 
the skills being learned in the course of professional devel- 
opment predicts the ultimate success of the reform effort 
(Cohen & Hill, 1998). 

• Teacher Skills. Skills are the classroom methods and 
techniques that are the basis of the planned instructional 
reform, such as reading comprehension strategies or 
problem-solving techniques. Assessing the teacher’s 



readiness to use a skill in the classroom is 
helpful, especially if the goal of the partic- 
ular professional development activity is to 
attain that level of skill. 

• Teacher Attitudes and Beliefs. A change 
in practice often depends upon a change in 
pedagogical attitudes or beliefs. Consider a 
school that wishes to promote cooperative 
learning. To properly implement this 
approach, teachers may need to believe 
that learning is a social activity and that 
students need to be enmeshed in a social 
network to articulate their thoughts and give one another 
feedback. If teachers adhere to a different belief about 
learning — for example, that learning is a process of acquir- 
ing isolated facts — they are unlikely to use cooperative 
learning effectively. For this reason, evaluating teacher atti- 
tudes and beliefs is helpful. 

How to Use the Information 

Participant learning from the professional development 
activity may be used in the following ways: 

• Clarify rationale. The attitudes and beliefs of teachers 
can identify areas of the project’s rationale or theory that 
need to be clarified. 

• Define next steps. One of the main functions in evaluat- 
ing learning is to identify what the next step in the profes- 
sional development should be. If there is a gap in the 
knowledge about certain concepts, arranging for additional 
readings or small study groups may help fill in the gaps. 

If some teachers have not yet acquired the skill to imple- 
ment a new technique, additional modeling or coaching 
may be arranged. 

• Refine training. Evaluating whether the participants 
learned what the professional development intended to 
teach confirms if the training has been successful and iden- 
tifies where it is lacking. If many participants report no 
growth in their knowledge or do not indicate an under- 
standing of the value of the program, the program has a 
need for improvement somewhere. 



The careful director 
of professional 
development will be 
sure to assess teacher 
learning continuously 
from a variety 
of sources. 
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How to Get the Information 

Fixed-Response Surveys. Surveys with fixed-response 
items are an indirect way to assess learning of knowledge 
and skills and changing of attitudes and beliefs. One way 
of approach to evaluating knowledge is to have partici- 
pants rate their preparedness to implement each skill in the 
classroom. One way of evaluating beliefs is to have the 
participants rate their agreement with various statements. 
(See the sample survey items at the bottom of the page.) 

Open-Ended Questions. The most direct way to assess 
learning after an event is with open-ended response items, 
as in the following examples: 

• “List two or three Big Ideas that you learned today about 
[insert TOPIC].” 

This question reveals what ideas clicked with the partici- 
pants. It is particularly helpful to find out if an idea didn’t 
click by virtue of its absence. 

• “List the questions that you still have about putting this 
information into practice.” 

Responses to this question help to determine next steps by 
identifying the unique concerns that teachers have. 

Observation of Simulated Lessons. Another way to 
assess participant learning is to observe teachers practicing 



Tip: Presurveys and Postsurveys 



One powerful technique of demonstrating 
change in teacher knowledge, skills, or attitudes 
is to administer the same survey twice — before 
and after the professional development service. 
Increases in self-ratings of skills or beliefs provide 
some evidence that the training program is working. 

newly learned skills in a simulated classroom experience. 
Whenever evaluating skill use through observation, it is 
necessary to have clear criteria for competent performance 
of the skill. These criteria need to be made explicit to the 
learners and should be modeled to them as well (Guskey, 
2000). For this same reason, it is useful to have an evalua- 
tion form that lists these criteria for the observer. 

Additional Resources for 
Evaluating Teacher Learning 

See “Evaluation Resources on the Web” on page 16 for 
links to additional examples of instruments that assess par- 
ticipant knowledge, skills, attitudes, and beliefs. 



Sample Survey Item for Rating Teacher Knowledge and Skill 


Please rate your 
level of knowledge 
and skill: 


Little 

knowledge 


Know of 
this, but am 
not prepared 
to put into 
practice 


Am now 
actively 
planning to 
implement 


Using the 
strategy, but 
working on 
logistical 
concerns 


Routinely 
use and 
satisfied 
with how 
it is going 


1 . Manage students 
engaged in project- 
based work 













Sample Survey Item for Rating Teacher Attitudes and Beliefs 


Statement 


Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Vocabulary knowledge is an important 
part of reading comprehension. 
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LEVELS 3 AND 4: EVALUATE ORGANIZATIONAL 
SUPPORT AND CLASSROOM IMPLEMENTATION 



E valuating organizational support and evaluating class- 
room implementation are intricately related. The 
amount and the effectiveness of organizational support 
affect how well teachers understand and apply new knowl- 
edge and skills, which in turn affects how well teachers 
implement their new learning in the classroom. Evaluating 
classroom implementation also identifies the components 
of an instructional reform on which teachers need addi- 
tional help. 

The role of the director of professional development at a 
school or district is to provide the organizational support 
for implementation. Such support comes in many forms, 
including resources, time, verbal encouragement, and orga- 
nizational policies. Without such support, implementation 
may falter — even if the professional developments events 
themselves achieved their goals for teacher learning. For 
this reason, Guskey (2000) and others advocate for an 
explicit evaluation of organizational support. Unfortunately, 
many instructional leaders and professional development 
directors find it difficult to directly evaluate their own 
organization, for a variety of reasons. To meet their needs, 
this section of the newsletter describes techniques for eval- 
uating classroom implementation in a manner that illumi- 
nates areas where organizational support is necessary. 

When observing implementation, it is necessary to deter- 
mine the key indicators of the proper use of a practice. 
Consider a school that is attempting to improve the guided 
reading component of its balanced literacy approach. 
Indicators of implementation may be the quality of teacher 
responses to reading errors and the appropriate use of 
learning centers. With highly specific indicators, it is pos- 
sible to ask specific questions and obtain meaningful data. 
These indicators, however, are useful only if the teachers 
understand them as well; otherwise, teachers will not have 
clarity about what constitutes correct classroom use of a 
new technique. The description of indicators depends on 
the level of the professional development. At early levels, 
it is useful to look at the outward signs of the attempt to 
adopt the practice. At a later level, the evaluator will want 
to focus on indicators of the quality of use. 



Topics to Investigate 

Assessments of implementation ask the following questions: 

• How frequently are teachers using the strategy? 

• How appropriately are teachers applying the strategy to 
the classroom? 

• Are teachers implementing all components of the partic- 
ular reform? 

• What sort of concerns do teachers have about 
implementation, and how can the school or district 
respond to those concerns? 

• Has the classroom environment changed to support the 
new methods or techniques? 

How to Use the Information 

From the perspective of the professional development 
director, the main purpose of evaluating implementation is 
to identify the components on which teachers require addi- 
tional training and the components on which they have 
achieved a degree of competency and regularity. The 
answers to the above questions primarily are used to start 
conversations with teachers about what they still need in 
order to implement the practice. By the same token, the 
mere fact that administrators are committed to evaluating 
implementation may be a motivating factor for teachers 
because it introduces accountability. Finally, evaluating the 
level of implementation helps to judge whether teachers 
are ready for the next stage of the overall professional 
development plan. 

How to Get the Information 

The following specific techniques can be used to evaluate 
classroom implementation: 

Hallway Walks. Hallway walks are brief classroom visits to 
observe the specific classroom behaviors that indicate proper 
implementation. The observer usually visits three to five 
classrooms in an hour. At this rate, data from the entire school 
or district can be collected over a week or two, giving the 
school leadership a good sense of the overall implementation. 
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The way to conduct a hallway walk is fairly simple. First, cre- 
ate a hallway- walk checklist listing the key indicators of the 
proper use of a practice. (See the sample checklist below.) 

Be sure to specify the teacher and student behaviors and 
classroom features that demonstrate those indicators. Next, 
schedule time for the principal or teacher leaders to conduct 
the hallway walks. Provide the principal with copies of the 
checklist to be used for the classrooms observed. 

Classroom teachers should be informed about their visitors in 
advance so they may coordinate a time to observe a particular 
lesson. Also, the evaluators should make clear to teachers that 
the observation is only to observe implementation and is not 
related to formal teacher evaluation. When all of the data is 
collected, it is useful to create a summary of the proportion of 
classrooms that adequately implemented each indicator. 

Classroom Observations. In contrast to hallway walks, a 
traditional classroom observation typically focuses on one 
teacher at a time. The point of the observation is to follow the 
progress of a teacher in learning a new technique, and to 



provide ongoing feedback and support. Notes & Reflections 
Issue 5, “Teacher Collaboration Supports Instmctional Change” 
provides tips and tools for doing teacher observations. It is 
available online (www.ncrel.org/info/notes/faU03/index.html). 

Lesson-Plan Observations. Viewing lesson-plan notebooks 
is an ideal complement to directly observing classrooms. 
These documents allow the evaluator to review entire weeks 
of lessons at a time. Such a perspective is useful for noticing 
ongoing patterns of instmction. For example, Ying Ying Chen, 
principal of Prairie Ridge Elementary School in Cedar 
Rapids, Iowa, examined one week of reading lesson plans for 
teachers who were implementing a balanced reading 
approach. This examination revealed an absence of lessons on 
word analysis. Principal Chen obtained additional assistance 
for her teachers in this area of instruction. 

Teacher Concerns. Teachers’ concerns about implement- 
ing the strategy are a guide for how to provide follow-up 
support. Typical stages of concern about implementation 
are specified in the Concems-Based Adoption Model 



Tool: Hallway-Walk Checklist 


K-1 

n=4 


2 

n=3 


3 

n=3 


4 

n=3 


5 

n=3 


Totals 

N=16 


Instructional Practices 

(less desired practices in italics) 


Plac 
teacht 
the pra 


e a tally r 
5f at the < 
:tice. Cot 
onl^ 


lark in ee 
)rade lev< 
nbine K v 
one K te 


ch cell for 
?l observe 
yith Gr. 1 i 
acher. 


each 
d using 
f there is 


Tally for all 
grades/N 














1. Individual self-selected reading 














2. Round-robin reading 














3. Whole-group phonics lesson 














4. Teacher holds writing conference 














5. Whole-group textbook lesson 
on mechanics 



Reproduced with permission from the School City of Hammond, Hammond, Indiana. 
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(Hord, Rutherford, Huling-Austin, & Hall, 1987). The 
model identifies three broad classes of concern (self, task, 
and impact), each of which may signal a different need of 
the teacher for the organization to support. Ruth Mueller, 
director of the Leadership Institute for School City of 
Hammond in Indiana, uses this model to understand what 
sort of support her teachers need. For exam- 
ple, teachers who are struggling to manage 
the new strategy — preparation and execu- 
tion — may require additional coaching and 
modeling and the opportunity to participate 
in study groups. Mueller states that “real 
conversations with teachers [provide] real 
evidence of what is going on. Listen to what 
they’re saying. They don’t even have to 
know you’re listening per se, because you 
sit in on workshops with them.” 

Professional development workshops and follow-up ses- 
sions are ideal opportunities to listen to teachers, especially 
the questions they ask of the presenter and the comments 
they make about their attempts to implement strategies in 
the classroom. Another opportunity to listen to teachers is 
when they discuss student work. The types of difficulties 
and student reactions that are presented during these ses- 
sions say a lot about each teacher’s level of concern about 
and extent of implementation. A third way to listen to 
teachers is to ask direct questions about their experiences. 
Principal Chen, for example, asks her teachers on a weekly 
basis, “What support do you need from us? What informa- 
tion do you need?” Scheduling more formal interviews with 
teachers is another way to find out about their concerns. 

Teacher Surveys. Teachers can rate their level of use for 
each component of the instructional reform. (A useful rating 
scale for teacher implementation is presented below, and a 
sample teacher survey is presented on page 12.) 



Evaluation experts at Learning Point Associates have noted 
that teacher surveys do not always give an accurate picture 
of the quality and extent of implementation, however. 
Teachers often have difficulty knowing if they are meeting 
the indicators for quality. For this reason, teacher surveys 
should serve as supplementary information, not the sole 
source of the evaluation. 



Additional Resource for Evaluating 
Organizational Support and 
Classroom Implementation 

Concerns-Based Adoption Model 

WWW. nas. edu/rise/backg4a. htni 

This Web page, provided by the National Academies, 
includes a chart that lists seven stages of concern, along 
with comments that teachers often make at each stage. 



“Real conversations with 
teachers [provide] real 
evidence of what is going 
on. Listen to what 
they ’re saying.” 

-Ruth Mueller 
School City of 
Hammond, Indiana 



The survey format also is a way to assess 
the adequacy of a school’s support for 
implementation of new techniques. Teacher 
feedback should be requested in this area. 

Student Work. The effective use of many 
kinds of instructional techniques will be 
directly reflected in student work. For exam- 
ple, the implementation of a mathematics 
curriculum that emphasizes problem solving 
will be demonstrated by students solving real-life, open- 
ended problems with a number of different strategies. 

Student Surveys. Another use of surveys is to ask students 
directly about how often they engage in certain classroom 
activities. These assessments can be used to track change 
over the course of the reform effort. (A sample student 
survey is presented on page 13.) 



Rating Scale for Teacher Implementation 


Strategy 


No plans to 
begin using 
yet. 


Am making 
plans to use 
soon. 


Have started 
using but not 
routinely. 


Use routinely. 


Use regularly 
and am 
comfortable 
modifying as 
needed. 


Strategy [x] 
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Tool: Sample Teacher Survey on Implementation of Strategic Reading 

The purpose of this survey is to gather information about the extent to which teachers have been implementing 
strategic reading in their classrooms. The results of this survey will be used to plan future professional development 
sessions. Please answer all questions as well as you can. 

1. How would you rate your general knowledge and experience with strategic reading? 

(Please check the one response that is most accurate for you.) 

□ Beginner/novice (have little or no knowledge and very limited experience) 

□ Intermediate (know the approach and have used it) 

□ Veteran (am very familiar with the approach and use regularly in my classroom) 

2. How many times have you worked with a strategic reading coach this year (the 2003-04 school year)? 

(Check the most accurate response.) 

□ Not yet □ 1-2 times □ 3-4 times □ 5-6 times □ More than 6 times 

3. Are you able to set aside regular time to plan how to incorporate strategic reading in your classroom? 

(Check one response.) 

□ Yes, definitely. □ No, not having planning time is a real problem. 

□ Not enough, but I still manage with the time I have. 

4. Please indicate your current level of use for each of the strategic reading strategies listed below. 



Strategy 


No plans to 
begin using 
yet. 


Am making 
plans to use 
soon. 


Have started 
using but not 
routinely. 


Use routinely. 


Use regularly 
and am 
comfortable 
modifying as 
needed. 


a. Think-aloud 












b. Predicting 












c. Summarizing 












d. Making inferences 












e. Graphic organizers 












f. Reciprocal teaching 













5. To what extent do you agree with the following statements? (Check one response per item.) 





Strongly 

Disagree 

1 


2 


Neither Agree 
nor Disagree 
3 


4 


Strongly 

Agree 

5 


a. My students are comfortable working in groups. 












b. My students are able to articulate their reading 
strategies. 












c. My students are able to use these reading 
strategies independently. 












d. Classroom management is difficult when 1 use 
strategic reading strategies. 












e. Lesson planning takes longer for 
strategic reading. 
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Tool: Sample Student Survey on Integrated Mathematics 



During the past school year, you have been a part of a pilot mathematics class called Integrated Mathematics. 

The Public School District is interested in learning what you think about this class — the types of instructional activi- 
ties that have taken place in this class as well as your attitudes towards math. 



In your mathematics lessons, how often are you asked to do the following? 

Never or 

Almost Never Some Lessons Most Lessons Every Lesson 

1. Explain the reasoning behind an idea 

2. Represent and analyze relationships using 
tables, charts, or graphs 

3. Work on problems for which there is no 
immediately obvious method of solution 

4. Work in class on projects and/or 
investigations that take a week or more 

5. Write equations to represent relationships 

6. Solve equations 

7. Practice computational skills 

8. How many times is math homework assigned to n 

Never Less than once 1-2 times 

9. When you are assigned mathematics homework, about how much time do you usually spend on it? (Circle your answer.) 

None 15 min. or less 15 to 30 minutes 30 min. to 1 hour More than 1 hour 



d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


d 


each week? (Circle your answer.) 




3-4 times 


Every day 







The following is a listing of ways teachers sometimes use to assess students (i.e., give them grades). Which of the 
following are used in your Integrated Mathematics class? Don't Know 







Yes 


No 


or Unsure 


10. Closed-choice tests (e.g., multiple choice, true/false) 


d 


d 


d 


1 1 . Essay or short-answer tests 




d 


d 


d 


12. Hands-on or performance assessments 




d 


d 


d 


13. Student assignments 




d 


d 


d 


14. Portfolios 




d 


d 


d 


15. Student Projects 




d 


d 


d 


1 6. Journals 




d 


d 


d 


For each of the following statements, mark the response that most closely represents your thinking. Don't 




Strongly 






Strongly Know or 




Agree 


Agree 


Disagree 


Disagree Unsure 



17. I enjoy learning math. d 

18. Math is important to everyone's life. 

19. I would like a job that involved 
using math. 

20. Math classes are more important 

for boys than girls. d 

21. Because of integrated mathematics, I plan 
on taking more math classes beyond 

my required credits. d 



d 

d 

d 

d 

d 



d 

d 

d 

d 

d 



d 

d 

d 

d 

d 



d 

d 

d 

d 

d 
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LEVEL 5: EVALUATE STUDENT LEARNING OUTCOMES 



A fter verifying that teachers are implementing their 
professional development learning, it is vital to con- 
firm that the new methods are having the intended effect 
on the students. Successful evaluation of student outcomes 
begins with three questions: 

• What are the criteria for success of the program? 

• What evidence will convincingly demonstrate these 
performance criteria? 

• How often is it desirable to evaluate these criteria? 

For the first question, evaluators must determine what 
behaviors of students indicate that they are learning in the 
desired way. In determining the criteria for success, it is 
crucial to specify a list of indicators for student perform- 
ance, just as it is necessary to specify indicators for suc- 
cessful implementation. These criteria should reflect the 
local and state standards for student achievement towards 
which schools must demonstrate progress. 

The answers to the second and third questions obviously 
depend on the specific goals of the program and the infor- 
mational needs of the evaluator. 

Topics to Investigate 

Questions about the impact on students should focus on 
the following areas: 

• Student Proficiency and Knowledge. Most goals for stu- 
dent learning aim to enhance proficiency for a certain 
skills or set of skills, such as reading comprehension, alge- 
braic formulas, or essay writing. Another highly valued 
student outcome is to improve academic knowledge in a 
specific content area, such as U.S. history or chemistry. 

• Engaged Learning. Another goal is student engagement 
in the learning task. Engagement refers to the way that stu- 
dents interact with the subject. Engaged students are ener- 
gized by their learning tasks, enjoy working on the topic, 
and set personal goals for their learning (Jones, Valdez, 
Nowakowski, & Rasmussen, 1994). 

• Student Beliefs and Attitudes. Many instructional 
reforms have the goal of changing student beliefs and atti- 
tudes about a subject. For example, an inquiry-based sci- 
ence program may have the goal of showing the relevance 
of science for every person. Another goal may be to alter 
students’ gender stereotypes about science. 



How to Use the Information 

The improvement of student learning is the ultimate pur- 
pose of any professional development effort. Demonstrating 
improvements in student learning provides feedback to 
teachers that their efforts to improve their teaching have 
paid off. Also, evidence of student progress toward learning 
goals demonstrates to parents and other stakeholders that 
the curricular and instructional approach is working and 
worth the money. If the evaluation does not demonstrate 
student improvement, the professional development director 
needs to consider ways to improve efforts. 

How to Get the Information 

Classroom Assessments. Assessments such as tests and 
quizzes, which teachers regularly use in their classrooms, 
often are well-aligned to the curriculum and provide an up- 
to-date picture of student performance. Yet these sources 
of data typically vary from classroom to classroom and 
year to year, making them unsuitable for pooling data from 
an entire school or district. Schools can overcome these 
limitations by creating a single series of tests for all class- 
rooms to use. The collaborative effort of teachers is needed 
to ensure that each test includes items that adequately sam- 
ple the topics in the curriculum, that questions are worded 
clearly, and that the grading criteria are well-defined. For 
these reasons, it is often desirable to field-test these assess- 
ments before their actual use (Killion, 2002). 

Student Portfolios. Another direct way to assess student 
learning is to periodically collect samples of each student’s 
work into a portfolio. Such portfolios allow teachers to 
view the progression of quality of the work over time. To 
this end, each work sample should be labeled by date as 
well as by type of task that it represents (e.g., a fictional 
story or an autobiographical essay). The frequency of col- 
lection may depend on how far along the professional 
development process is. When the implementation is just 
beginning, a weekly examination of student work may be 
useful to determine whether the new classroom techniques 
are having the desired impact on students. If the imple- 
mentation has stabilized, the frequency of analyzing port- 
folios may decrease to once per quarter or semester. 

To be useful for evaluation of student progress, the work 
collected in portfolios needs to be interpreted. For exam- 
ple, teachers can participate in study groups that examine 
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student work and identify overall areas of strength and 
weakness. Another use of the portfolios is to demonstrate 
educational progress to parents and students. Portfolios 
also provide an opportunity to involve students and parents 
in the assembly of the portfolio by asking them to decide 
which samples of student work should be chosen. 

Student Surveys. One way to evaluate student engagement 
is to ask students directly about their enjoyment of the topic 
and the degree to which they are interested in the tasks they 
do as part of their coursework. Attitudes and beliefs also 
can be evaluated in this fashion, with questions that address 
students’ perceived competence in the course, beliefs about 
the importance of the subject, and other questions of inter- 
est. Finally, students can rate their own perceptions about 
how much they have learned. The “Sample Student Survey 
on Integrated Mathematics” (page 13) illustrates these 
items. 

School Records. School records often are helpful for 
demonstrating student engagement. Examples include 
attendance records for particular classes, rate of assign- 



ment completion, and percentage of students who com- 
plete a certain academic level, such as advanced algebra. 

Additional Resources for Evaluating 
Student Outcomes 

Kuhs, T. M. (1997). Measure for measure: Using 
portfolios in K—8 mathematics. Portsmouth, NH: 
Heinemann. 

Stiggins, R. (1997). Student centered classroom 
assessments (2nd ed.). Columbus, OH: Merrill. 

Wiggins, G., & McTighe, J. (1998). Understanding by 
design. Alexandria, VA: Association for Supervision 
and Curriculum Development. 

Williams, J. (2000). Implementing portfolios and student- 
led conferences. ENC Focus, 7(2), 21-23. Retrieved 
February 17, 2004, from 
http://www.enc.org/features/focus/archive/ 
assessment/document. shtm?input=FOC-00 1560-index 



ORGANIZING THE EVALUATION 



B efore beginning the evaluation, it is crucial to plan all 
of its steps to ensure that no opportunity to collect 
information is lost. (See the “Sample Evaluation 
Organizer” below.) For each level of the evaluation, it is 
necessary to specify the following information: 

• Evaluation information and data that are needed: 

What questions do we want answered? 



• Data collection method: How will the information 
be gathered? 

• Who is responsible: Who will develop the instruments, 
administer them, analyze data, and write the report? 

• What resources are needed: How much time, money, 
and staff time are necessary? 



Tool: Sample Evaluation Organizer 


Evaluation Level 


Evaluation 
Information and 
Data Needed 


Data Collection 
Method 


Responsibility 


Resources 


Level 1 : Teacher 
Reactions 


Did teachers think 
the workshop 
went well? 


End-of-workshop 

surveys 


Rachel 


2 hours of 
staff time 


Level 2: Teacher 
Learning 


Will the district 
provide planning 
time? 


Content survey 


Holly 


12 hours to create 
survey and analyze 


Level 3: 
Organizational 
Support 


Did they learn the 
content? 


Will know if they 
do it 


Jocelyn will speak 
to a sample of 
teachers 


2 hours 


Level 4: 
Classroom 
Implementation 


Are teachers using 
the new strategy in 
the classroom? 


Use a common 
observation 
protocol 


Jim will lead a 
group of five 
teachers 


Subs for teachers 
doing observations 


Level 5: 

Student Learning 
Outcomes 


Did test scores 
go up? 


Via test results 
in June 


N/A; Jim will dis- 
cuss test results 


3 hours of 
preparation time 
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Evaluation Resources on the Web 


Standards 

National Staff Development Council 

www.nsdc.org 

The National Staff Development Council offers Standards for Staff 
Development (www.nsdc.org/standards/index.cfm), and a Self-Assessment and 
Planning Tool (www.nsdc.org/standards/about/assessment.pdf) for assessing the 
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Planning and Organizing the Evaluation 
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WWW. oerl.sri. com/modide/mod4/m4 _pl.html 

The Online Evaluation Resource Library offers a step-by-step guide to design- 
ing an evaluation. 

Evaluating Whole-School Reform Efforts: A Guide for District and 
School Staff 

WWW. nwrac. org/whole-school/index. litml 

This guidebook from the Northwest Regional Educational Laboratory provides 
extensive advice for conducting evaluations of program implementation and 
impact. 


Journal of Staff Development 

www.nsdc.org/library/publications/jsd/ 

The Fall 2003 issue of the Journal of Staff Development is devoted to evalua- 
tion. It includes an article by Joellen Killion titled “8 Smooth Steps: Solid 
Footwork Makes Evaluation of Staff Development Programs a Song” 
(www.nsdc.org/library/publications/jsd/jkillion244.cfm). Also, see the Fall 1998 
article by Thomas Guskey titled “The Age of Our Accountability: Evaluation 
Must Become an Integral Part of Staff Development” 
(www.nsdc.org/library/publications/jsd/guskeyl94.cfm). 

Instruments 

Local Systemic Change (LSC) Instruments 

www.horizon-research.coni/LSC/manual/0304/ 

This Web site contains all of the instruments used by Horizon Research Inc. to 
evaluate the Local Systemic Change initiative of the National Science 
Foundation. The instruments include surveys, interview protocols, and observa- 
tion protocols. 

Online Evaluation Resource Library (OERL) Evaluation Instruments for 
Teacher Education Projects 

WWW. oerl.sri. com/instruments/te/ 

The Online Evaluation Resource Library presents numerous examples of evalu- 
ation instruments, including surveys, interview protocols, observation protocols, 
and assessment of content knowledge. 


Evaluation Resources in Print 
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for Supervision and Curriculum Development. 

Guskey, T. R. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin 
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evaluation. Newbury Park, CA: Corwin Press. 
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Staff Development Council. 
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